Think Globally, Act Locally: An Introduction to Domain-based Parallelism and Problem Decomposition Methods

“Think globally; act locally.” This bumper sticker maxim has a lot to say to practitioners of contemporary high performance computing. It is increasingly incumbent on computational scientists to respect the data access hierarchies that accompany the large memories required by applications programs. These hierarchies are imposed, ultimately, by the finite size of data storage media and the finite speed of light, but their presence is asserted more immediately by the hardware and software overheads of system protocols for the delivery of data. From the frame of reference of any given processing element, an approximate cost function can be constructed for the minimum time required to access a memory element that is any given logical or physical distance away. Such cost functions typically consist of plateaus separated by sharp discontinuities that correspond to software latencies where some boundary of the hierarchy, such as a cache size or a local memory size, is crossed. The ratio of times required to access remote and local data varies from 10 to 10 in typical architectures, the latter being characteristic of network cluster computing. An underlying motivation for the development of problem decomposition algorithms is that these discontinuities should explicitly be respected by user applications. If users cannot afford to treat memory as “flat” in large problems, then neither can they afford to treat all nonzero data dependencies on an equal footing. Consequently, algorithms must adapt to architecture, guided by knowledge of the relative strengths of different couplings from the underlying physics. Ironically, such forced adaptation sometimes results not in compromise, but in the discovery of intrinsically better methods for flat memory environments, as well. Steady-state natural and human-engineered systems are often zero-sum networks in which the overall distribution of a quantity to be determined is con∗Department of Computer Science, Old Dominion University, Norfolk, VA 23529-0162 and Institute for Computer Applications in Science and Engineering, NASA Langley Research Center, Hampton, VA 23681-0001. email: [email protected]. †Department of Computer Science, University of Minnesota, Minneapolis, MN 55455-0154. email: [email protected]. ‡Minnesota Supercomputer Institute, 1200 Washington Ave. S., Minneapolis, MN 55415. email: [email protected].